Progressive growing of self-organized hierarchical representations for exploration

Designing agent that can autonomously discover and learn a diversity of structures and skills in unknown changing environments is key for lifelong machine learning. A central challenge is how to learn incrementally representations in order to progressively build a map of the discovered structures and re-use it to further explore. To address this challenge, we identify and target several key functionalities. First, we aim to build lasting representations and avoid catastrophic forgetting throughout the exploration process. Secondly we aim to learn a diversity of representations allowing to discover a "diversity of diversity" of structures (and associated skills) in complex high-dimensional environments. Thirdly, we target representations that can structure the agent discoveries in a coarse-to-fine manner. Finally, we target the reuse of such representations to drive exploration toward an "interesting" type of diversity, for instance leveraging human guidance. Current approaches in state representation learning rely generally on monolithic architectures which do not enable all these functionalities. Therefore, we present a novel technique to progressively construct a Hierarchy of Observation Latent Models for Exploration Stratification, called HOLMES. This technique couples the use of a dynamic modular model architecture for representation learning with intrinsically-motivated goal exploration processes (IMGEPs). The paper shows results in the domain of automated discovery of diverse self-organized patterns, considering as testbed the experimental framework from Reinke et al. (2019)

The Gamma-Ensemble - Adaptive Reinforcement Learning via Modular Discounting

Reinforcement learning allows artificial agents to learn complex tasks, such as playing Go on an expert level. Still, unlike humans, artificial agents lack the ability to adapt learned behavior to task changes, or to new objectives, such as to capture as many opponent pieces within a given number of moves, instead of simply winning. The Independent Gamma-Ensemble (IGE), a new brain-inspired framework, allows such adaptations. It is composed of several Q-learning modules, each with a different discount factor. The off-policy nature of Q-learning allows modules to learn several policies in parallel, each representing a different solution for the payoff between a high reward sum and the time to gain it. The IGE adapts to new task conditions by switching between its policies (transfer learning). It can also decode the expected reward sum and the required time for each policy, allowing it to immediately select the most appropriate policy for a new task objective (zero-shot learning). Additionally, this allows to optimize the average reward in discrete MDPs where non-zero reward is only given in goal states. The convergence to the optimal policy can be proven for such MDPs. The modular structure behind the IGE can be combined with many reinforcement learning algorithms and applied to various tasks, allowing to improve the adaptive abilities of artificial agents in general.Okinawa Institute of Science and Technology Graduate Universit

Learning ‘‘graph-mer’’ Motifs that Predict Gene Expression Trajectories in Development

A key problem in understanding transcriptional regulatory networks is deciphering what cis regulatory logic is encoded in gene promoter sequences and how this sequence information maps to expression. A typical computational approach to this problem involves clustering genes by their expression profiles and then searching for overrepresented motifs in the promoter sequences of genes in a cluster. However, genes with similar expression profiles may be controlled by distinct regulatory programs. Moreover, if many gene expression profiles in a data set are highly correlated, as in the case of whole organism developmental time series, it may be difficult to resolve fine-grained clusters in the first place. We present a predictive framework for modeling the natural flow of information, from promoter sequence to expression, to learn cis regulatory motifs and characterize gene expression patterns in developmental time courses. We introduce a cluster-free algorithm based on a graph-regularized version of partial least squares (PLS) regression to learn sequence patterns—represented by graphs of k-mers, or “graph-mers”—that predict gene expression trajectories. Applying the approach to wildtype germline development in Caenorhabditis elegans, we found that the first and second latent PLS factors mapped to expression profiles for oocyte and sperm genes, respectively. We extracted both known and novel motifs from the graph-mers associated to these germline-specific patterns, including novel CG-rich motifs specific to oocyte genes. We found evidence supporting the functional relevance of these putative regulatory elements through analysis of positional bias, motif conservation and in situ gene expression. This study demonstrates that our regression model can learn biologically meaningful latent structure and identify potentially functional motifs from subtle developmental time course expression data

Progressive growing of self-organized hierarchical representations for exploration

International audienceDesigning agent that can autonomously discover and learn a diversity of structures and skills in unknown changing environments is key for lifelong machine learning. A central challenge is how to learn incrementally representations in order to progressively build a map of the discovered structures and re-use it to further explore. To address this challenge, we identify and target several key functionalities. First, we aim to build lasting representations and avoid catastrophic forgetting throughout the exploration process. Secondly we aim to learn a diversity of representations allowing to discover a "diversity of diversity" of structures (and associated skills) in complex high-dimensional environments. Thirdly, we target representations that can structure the agent discoveries in a coarse-to-fine manner. Finally, we target the reuse of such representations to drive exploration toward an "interesting" type of diversity, for instance leveraging human guidance. Current approaches in state representation learning rely generally on monolithic architectures which do not enable all these functionalities. Therefore, we present a novel technique to progressively construct a Hierarchy of Observation Latent Models for Exploration Stratification, called HOLMES. This technique couples the use of a dynamic modular model architecture for representation learning with intrinsically-motivated goal exploration processes (IMGEPs). The paper shows results in the domain of automated discovery of diverse self-organized patterns, considering as testbed the experimental framework from Reinke et al. (2019)

Sex-Associated Differences in Cytomegalovirus Prevention: Prophylactic Strategy is Potentially Associated With a Strong Kidney Function Impairment in Female Renal Transplant Patients

Post-transplantation cytomegalovirus (CMV) syndrome can be prevented using the antiviral drug (val)ganciclovir. (Val)ganciclovir is typically administered following a prophylactic or a pre-emptive strategy. The prophylactic strategy entails early universal administration, the pre-emptive strategy, early treatment in case of infection. However, it is not clear which strategy is superior with respect to transplantation outcome; sex-specific effects of these prevention strategies are not known. We have retrospectively analyzed 540 patients from the multi-centre Harmony study along eight pre-defined visits: 308 were treated according to a prophylactic, 232 according to a pre-emptive strategy. As expected, we observed an association of prophylactic strategy with lower incidence of CMV syndrome, delayed onset and lower viral loads compared to the pre-emptive strategy. However, in female patients, the prophylactic strategy was associated with a strong impairment of glomerular filtration rate one year post-transplant (difference: -11.8 ± 4.3 ml min-1·1.73 m-2, p = 0.006). Additionally, we observed a tendency of higher incidence of acute rejection and severe BK virus reactivation in the prophylactic strategy group. While the prophylactic strategy was more effective for preventing CMV syndrome, our results suggest for the first time that the prophylactic strategy might lead to inferior transplantation outcomes in female patients, providing evidence for a strong association with sex. Further randomized controlled studies are necessary to confirm this potential negative effect

Risk factors for Epstein–Barr virus reactivation after renal transplantation: Results of a large, multi‐centre study

Epstein-Barr virus (EBV) reactivation is a very common and potentially lethal complication of renal transplantation. However, its risk factors and effects on transplant outcome are not well known. Here, we have analysed a large, multi-centre cohort (N = 512) in which 18.4% of the patients experienced EBV reactivation during the first post-transplant year. The patients were characterized pre-transplant and two weeks post-transplant by a multi-level biomarker panel. EBV reactivation was episodic for most patients, only 12 patients showed prolonged viraemia for over four months. Pre-transplant EBV shedding and male sex were associated with significantly increased incidence of post-transplant EBV reactivation. Importantly, we also identified a significant association of post-transplant EBV with acute rejection and with decreased haemoglobin levels. No further severe complications associated with EBV, either episodic or chronic, could be detected. Our data suggest that despite relatively frequent EBV reactivation, it had no association with serious complications during the first post-transplantation year. EBV shedding prior to transplantation could be employed as biomarkers for personalized immunosuppressive therapy. In summary, our results support the employed immunosuppressive regimes as relatively safe with regard to EBV. However, long-term studies are paramount to support these conclusions

High incidence and viral load of HHV-6A in a multi-centre kidney transplant cohort

Human herpesvirus 6 (HHV-6) is a common opportunistic pathogen in kidney transplant recipients. Two distinct species of HHV-6, HHV-6A and HHV-6B, have been identified, of which the latter seems to be dominant. However, it is unclear whether they increase the likelihood of other viral reactivations. We characterized a multi-centre cohort of 93 patients along nine study visits for viral load. We tested for the following viruses: HHV-6A and HHV-6B, the herpesviruses cytomegalovirus (CMV) and Epstein-Barr virus (EBV) and the polyomavirus BK (BKV). We detected HHV-6A viral load in 48 (51.6%) patients, while the incidence of HHV-6B was much lower, being detected in 6 (6.5%) patients. The incidence of HHV-6A was higher than of BKV, CMV and EBV. HHV-6A also demonstrated higher viral loads than the rest of viruses. There was a non-significant trend of association between HHV-6A and HHV-6B as co-infection, whereas no increased incidence of other viruses among patients with HHV-6A reactivation was observed. There was no negative effect of high HHV-6A (>10,000 copies/ml) load on markers of renal graft and hepatic function or blood count twelve months post-transplant. In contrast to previously published data, our results show a clear dominance of HHV-6A in peripheral blood when compared to HHV-6B, with higher incidence and viral load levels. Despite the high HHV-6A loads observed, we did not identify any negative effects on posttransplant outcome

Facilitating Memory for Novel Characters by Reducing Neural Repetition Suppression in the Left Fusiform Cortex

Gui Xue is with Beijing Normal University and University of Southern California, Leilei Mei is with Beijing Normal University and University of California Irvine, Chuansheng Chen is with University of California Irvine, Zhong-Lin Lu is with University of Southern California, Russell A. Poldrack is with UT Austin, Qi Dong is with Beijing Normal University.Background -- The left midfusiform and adjacent regions have been implicated in processing and memorizing familiar words, yet its role in memorizing novel characters has not been well understood. Methodology/Principal Findings -- Using functional MRI, the present study examined the hypothesis that the left midfusiform is also involved in memorizing novel characters and spaced learning could enhance the memory by enhancing the left midfusiform activity during learning. Nineteen native Chinese readers were scanned while memorizing the visual form of 120 Korean characters that were novel to the subjects. Each character was repeated four times during learning. Repetition suppression was manipulated by using two different repetition schedules: massed learning and spaced learning, pseudo-randomly mixed within the same scanning session. Under the massed learning condition, the four repetitions were consecutive (with a jittered inter-repetition interval to improve the design efficiency). Under the spaced learning condition, the four repetitions were interleaved with a minimal inter-repetition lag of 6 stimuli. Spaced learning significantly improved participants' performance during the recognition memory test administered one hour after the scan. Stronger left midfusiform and inferior temporal gyrus activities during learning (summed across four repetitions) were associated with better memory of the characters, based on both within- and cross-subjects analyses. Compared to massed learning, spaced learning significantly reduced neural repetition suppression and increased the overall activities in these regions, which were associated with better memory for novel characters. Conclusions/Significance -- These results demonstrated a strong link between cortical activity in the left midfusiform and memory for novel characters, and thus challenge the visual word form area (VWFA) hypothesis. Our results also shed light on the neural mechanisms of the spacing effect in memorizing novel characters.This study was supported by the Program for New Century Excellent Talents in University, the National Science Foundation (grant numbers BCS 0823624 and BCS 0823495), the National Institute of Health (grant number HD057884-01A2), and the 111 Project of China (B07008). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Psycholog